Methods and apparatus for autonomous robotic control

ABSTRACT

Sensory processing of visual, auditory, and other sensor information (e.g., visual imagery, LIDAR, RADAR) is conventionally based on “stovepiped,” or isolated processing, with little interactions between modules. Biological systems, on the other hand, fuse multi-sensory information to identify nearby objects of interest more quickly, more efficiently, and with higher signal-to-noise ratios. Similarly, examples of the OpenSense technology disclosed herein use neurally inspired processing to identify and locate objects in a robot&#39;s environment. This enables the robot to navigate its environment more quickly and with lower computational and power requirements.

CROSS-REFERENCE TO RELATED PATENT APPLICATION

This application is a continuation of U.S. application Ser. No.15/463,299, filed on Mar. 20, 2017, and entitled “Methods and Apparatusfor Autonomous Robotic Control,” which is a continuation of Ser. No.14/662,657, filed Mar. 19, 2015, now U.S. Pat. No. 9,626,566, andentitled “Methods and Apparatus for Autonomous Robotic Control,” whichclaims priority, under 35 U.S.C. § 119(e), from U.S. Application No.61/955,756, filed Mar. 19, 2014, and entitled “Methods and Apparatus forAutonomous Robotic Control.” Each of these applications is herebyincorporated herein by reference in its entirety.

GOVERNMENT SUPPORT

This invention was made with government support under Contract No.FA8750-12-C-0123 awarded by Air Force Research Laboratory (AFRL), underContract No. NNX12CG32P awarded by NASA Phase I STTR, and under ContractNo. NNX13CL63C awarded by NASA Phase II STTR. The government has certainrights in the invention.

BACKGROUND

For a mobile robot to operate autonomously, it should be able to locateobstacles and steer around them as it moves within its environment. Forexample, a mobile robot may acquire images of its environment, processthem to identify and locate obstacles, then plot a path around theobstacles identified in the images. In some cases, a mobile robot mayinclude multiple cameras, e.g., to acquire stereoscopic image data thatcan be used to estimate the range to certain items within its field ofview. A mobile robot may also use other sensors, such as radar or lidar,to acquire additional data about its environment. Radar is particularlyuseful for peering through smoke or haze, and lidar returns cansometimes be used determine the composition of objects within theenvironment. A mobile robot may fuse lidar, radar, and/or other datawith visible image data in order to more accurately identify and locateobstacles in its environment. To date, however, sensory processing ofvisual, auditory, and other sensor information (e.g., LIDAR, RADAR) isconventionally based on “stovepiped,” or isolated processing, withlittle interaction between modules.

SUMMARY

Embodiments of the present invention include a system for identifyingand locating objects in a robot's environment. Such a system may includean image sensor and processor operably coupled to the image sensor. Inoperation, the image sensor acquires a plurality of images of at least aportion of the environment surrounding the robot. The processortranslates each image in the plurality of images from the image sensor'sframe of reference to an allocentric frame of reference. The processoridentifies a position, in the allocentric frame of reference, of anobject appearing in at least one image in the plurality of images. Andthe processor determines if the object appears in at least one otherimage in the plurality of images based on the position, in theallocentric frame of reference, of the object.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein. In particular, allcombinations of claimed subject matter appearing at the end of thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein. It should also be appreciated that terminologyexplicitly employed herein that also may appear in any disclosureincorporated by reference should be accorded a meaning most consistentwith the particular concepts disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The skilled artisan will understand that the drawings primarily are forillustrative purposes and are not intended to limit the scope of theinventive subject matter described herein. The drawings are notnecessarily to scale; in some instances, various aspects of theinventive subject matter disclosed herein may be shown exaggerated orenlarged in the drawings to facilitate an understanding of differentfeatures. In the drawings, like reference characters generally refer tolike features (e.g., functionally similar and/or structurally similarelements).

FIG. 1 is a block diagram of an example OpenEye system.

FIG. 2 is a block diagram of the Where Pathway module shown in FIG. 2.

FIG. 3 is a block diagram of the What Pathway module shown in FIG. 2.

FIG. 4 is a block diagram of an alternative classifier architecturesuitable for implementing the view layer and the object layer in theWhat Pathway module shown in FIG. 3.

FIG. 5 illustrates control of a robot using the OpenEye system via aremote controller, such as a tablet or smartphone.

FIG. 6 illustrates a process for identifying and locating objects in arobot's environment by fitting a spatial shroud to successive images ofthe robot's environment.

FIGS. 7A-7C show fitting a spatial shroud to an object in differentimages of the robot's environment.

DETAILED DESCRIPTION

The methods described herein provide an exemplary unified technology foridentifying, learning, localizing, and tracking objects based on camera(e.g., RGB) input. Some examples of this technology are called “OpenEye”and can be implemented as an artificial, active sensory system and aunified framework for processing sensor data, including but not limitedto image data. OpenEye may be utilized in both artificial (e.g.,simulated environments, such as an environment generated syntheticallyvia a video-game engine) and natural environments (e.g., an environmentexperienced by an unmanned aerial, ground, or submersible vehicle).

In operation, OpenEye learns incrementally about its visual input, andidentifies objects in the sensor field of view and categorizes thoseobjects by identity and position. OpenEye can operate with or withoutsupervision, and does not require a manual labeling of object ofinterest to learn object identity. OpenEye can also accept user input toverbally label objects.

OpenEye simulates mammalian brains' dorsal (where—controlling where tolook) and ventral (what—controlling the content of the image) pathwaysby using simulated eye movements (in virtual or real cameras) to learnidentity of objects in complex images (see, e.g., Mishkin andUngerleider 1982 and Webster et al., 1994).

In some implementations OpenEye uses a space-variant, log-polarrepresentation of the input visual field to sample the image “view”generated by each eye movement. The log-polar representation providessome invariance to translation/rotation, and substantial savings inprocessing time with better scalability to large datasets by employingnon-uniform input sampling and rapid scan of image segments, as opposedto processing the whole image at uniform resolution (Traver andBernardino, 2010). The model uses the what-to-where feedback to samplethe image intelligently. OpenEye does so by using the knowledge of theidentity of the current object and its context to focus on spatiallocations that yield greatest disambiguation of competing objectidentity (e.g., areas of an image that are more unique to an object).OpenEye may be validated on natural and synthetic images, as well as onthe standard MNIST handwritten digit dataset.

As opposed to other approaches (e.g., neural networks), the OpenEyesystem may not rely on extensive training (batch training) to be able toclassify correctly objects in the data stream, and can learn newknowledge online (i.e., during performance) without corrupting orforgetting previously learned knowledge. Additionally, the system isable to autonomously search for information in an image via an activevisual search process, which mimics the mechanism used by mammals torapidly and efficiently scan their visual world for importantinformation. OpenEye memory system is designed to allow on-lineoptimization of synaptic memories. Additionally, OpenEye can mimic humaneye movements by reproducing human fixation patterns with or without atraining session where OpenEye learns the fixation location of a humanuser via eye-tracker.

Neurally Inspired Robot Perception, Object Identification, and ObjectLocation

A conventional robot does not perceive its environment like a human. Forexample, a robot may “see” its environment by acquiring imagery of someor all or its environment at a uniform resolution. It then processes theimagery by dividing the imagery into a grid of pixels and examining eachpixel in the grid. This process can take too much time and too muchenergy to be useful for identifying objects moving relative to therobot, especially if the robot is moving at relatively high velocity(e.g., a drone flying at low altitude). In addition, the robot may spendan inordinate amount of time processing empty or irrelevant pixels.

A human does not process the detail of entire images on a pixel-by-pixelbasis. Instead, the human eye acquires imagery of non-uniformresolution: the central part of the retina, or fovea, which is denselypacked with light-sensitive cones, acquires the central part of eachimage at relatively fine resolution. And the peripheral portion of theretina, which is covered at lower density with light-sensitive rods andcones, acquires the peripheral portion of each image at coarserresolution. The resulting “foveated imagery” has resolution that variesspatially across each image, with the finest resolution at a fixationpoint and coarser resolution elsewhere. This notion of obtaining imageryat a resolution that varies spatially across each image is referred toherein as “foveation.”

To account for the spatial variation in image resolution, a human moveshis or her eyes rapidly among different points in his or her field ofview. For instance, a human may fixate on points at or near aninteresting portion of a scene, such as a face, for relatively longperiods, and fixate on points at or near less interesting portions ofthe scene, such as a tree, for shorter periods, if at all. These quick,simultaneous movements to different fixation points, or saccades, allowa human to identify and locate items of interest without spending timeor energy examining interesting portions of the scene.

Similarly, the OpenEye technology disclosed herein allows a robot toidentify and locate objects in its environment using “foveated” datacollection and “saccade” style imaging as explained below with respectto FIGS. 1-4. For instance, one or more processors may controlcollection and processing of visual imagery according to a neural modelinspired by the human brain. A camera or other sensor acquires imageryof the robot's environment and passes this imagery to a graphicsprocessing unit (GPU) or other suitable processor, which locates andidentifies one or more objects in the imagery (e.g., using the What andWhere pathways described in greater detail below) based on the imageryitself and information about the sensor's orientation, position, and/orfield of view. In some cases, the GPU may translate the imagery amongdifferent frames of reference, including camera-centered, robot-basedegocentric, and allocentric frames of reference, to make processing moreefficient and/or more precise.

The processor also determines the next fixation point of the sensorsystem based on the location and/or identity of the object(s). In somecases, it transmits movement vector representing the saccade between thecurrent fixation point and the next fixation point to an actuator thatthen actuates the sensor appropriately. For instance, the processor maycause a pan-tilt actuator to move a camera mounted on the robot so as toacquire imagery of an object from different angles and/or positions. Therobot itself may move to change the sensor's field of view. In othercases, the processor may cause synthetic “saccades,” e.g., by processingdifferent portions of the same image or different portions of differentimages at different resolutions depending on the objects and theirlocations. The robot may also use object information and sensor positionand orientation data to inhibit the sensor from fixating repeatedly onthe same object or the same portion of the scene.

Because the technology disclosed herein mimics human neural processing,it can process imagery and other sensory data more efficiently andidentify objects in the robot's environment more quickly. This isespecially useful for robots in hazardous applications, such asplanetary exploration, where processing and battery efficiency arecritical, and for robots that collect large volumes of data, suchsurveillance drones, where efficient sense-making is key to interpretinglarge amounts of real-time data. It also has general application to alltypes of vision systems, including simulations, such as those used invideo games, flight simulators, etc.

Visual Stream Exploration and Visual Object Learning

The OpenEye model proposes a method for combining visual streamexploration and visual object learning. Each is considered below.

Visual Stream Exploration Models

The computational model proposed by Itti and Koch (2001) simulates anaspect of human vision that predicts the probability that a certainimage area will attract an observer's attention and eye movements. Themodel only includes bottom-up, or sensory features, while OpenEye alsoaccounts for cognitive (top-down) biases on eye movements. Additionally,the model does not include learning, object, or scene recognition, whichare instead incorporated in OpenEye, where they bias image streamexploration as discussed below.

OpenEye also differs from Riesenhuber and Poggio's (1999) neural model,which employs a spatially homogenous representation of the image vs.OpenEye's spatially variant representation and use of sensor movement.Both the Itti & Koch (2001) and Riesenhuber & Poggio (1999) modelspostulate that visual objects need to be identified in one glance.OpenEye, instead, accounts for the potential need to explore the inputsensory image to gather additional evidence for recognition, which isparticularly useful for ambiguous objects/scenes (e.g., occludedobjects).

Visual Object Learning Models

In terms of learning, OpenEye may use two interchangeable learningmethodologies. The first method is based on the Baraldi and Alpaydin(1998, 2002) and Baraldi and Parmiggiani (1997) learning models, whichprovide the following benefits. The second method is based on arecurrent adaptive architecture. Both methodologies simultaneouslyimplement fast and slow learning.

Usually, fast learning (e.g., Carpenter and Grossberg, 1987) systemsunderperform slow-learning ones (Rumelhart et al., 1986), but the formerare much more useful in engineered system such as robots or sensorsoperating in real-time in a rapidly changing environment. After onlysingle instance of presentation of each item, humans and other animalscan learn to recognize pictures, words, names, and faces, and recordingat a local cellular level confirms that neurons can change to reflectsuch fast learning (Bunzeck & Düzel, 2006; Rutishauser et al., 2006). Todate, no artificial system has been engineered to achieve this goal in amachine.

Several object recognition algorithms have been developed over the lastfew decades (for reviews, see Besl and Jain, 1985; Logothetis andSheinberg, 1996; Riesenhuber and Poggio, 2000; Bengio et al., 2012). Ingeneral, a commonality between these algorithms is the focus on findingthe appropriate representation for the data, where the difference amongalgorithms performance is due to the nature of the features/input datatransformations. For instance, convolutional network models (Ranzato etal., 2007; Jarrett et al. 2009; LeCun et al., 2010) and restrictedBoltzmann machines (Smolensky, 1986; Salakhutdinov and Hinton, 2009) areamong the best object recognition algorithms. Both classes of algorithmsperform three main steps:

a) feature extraction. This can be either hardwired, random, or learned;b) non-linear transformation on the resulting filtered data; andc) A pooling step on the result of step b). The connectivity betweenstages and the number of filter-transform-pool stages can vary.

Deep learning networks include networks where there are several layersof stacked filter-transform-pool, e.g. in the HMAX model (Riesenhuber &Poggio, 1999) and deep belief networks (Hinton et al., 2006).

Similarly, Spratling (2008, 2009, 2012) has introduced severalrecognition systems built of stackable “cortical” modules. These modelsare composed of modules that work hierarchically and perform a processcalled “predictive coding”, that looks very akin to matching in an ARTsystem. A close examination of the derivation of the learning laws inthese systems (Spratling et al., 2009) reveals that they were developedas an incremental version of a well-known batch coding algorithm,non-negative matrix factorization (NMF), developed by Lee and Seung(1997, 1999). The algorithm presented by Spratling at al. does allowincremental (fast) learning, but does not include methods for objectsegregation/segmentation, scene recognition, and active vision.

However, none of the above-mentioned object recognition algorithms dealswith the issues of how objects are separated from their background, andneither of those models uses space-variant sampling.

The ARTScan (Fazl et al., 2009) model, the saccading restrictedBoltzmann machine (sRBM) (Larochelle & Hinton, 2012), and the entropyminimization algorithm of saccades (Friston et al., 2012)

The saccading restricted Boltzmann machine (Larochelle and Hinton, 2012)uses space variant vision. However, it does not include a mechanism thatinforms the system when the system stops fixation from an object andstarts fixating on another, which is provided by a human supervisor. Thesystem could not tell apart two identical objects presented side-by-sidewith a spatial gap separating them.

The entropy minimization algorithm of saccades (Friston et al., 2012)includes bi-directional What-to-Where stream interactions but does notuse space-variant vision, and it suffers from the same issue asLarochelle and Hinton (2012) in terms of object fixation memory.

The ARTScan (Fazl et al., 2009) model includes Where-to-What interactionin guiding when the What system should learn/stop learning, but does notinclude What-to-Where interactions to inform eye movement and visualsearch. Additionally, OpenEye differs from ARTScan in these additionaldimensions:

OpenEye and ARTScan use a different log-polar sampling;

OpenEye shroud formation is feed-forward;

OpenEye is designed to operate in 3D environments in a noisy background;

OpenEye is designed to handle self-motion;

OpenEye employs a concept of temporal continuity to support dynamicscenes;

OpenEye can combine multiple saliencies, endogenous spatial attention,attention to specific features in order to make next saccade; and

While ARTScan used handcrafted images OpenEye can be used with arbitraryimage data, such as the standard MNIST database;

Object learning models from Baloch and Waxman (1991), Bradski andGrossberg, (1995), Seibert and Waxman (1992) do use space-varianttransformation, or “cortical magnification”, but only focus staticallyat an object's center-of-mass.

OpenEye methods discussed in Sections 4 employ a learning scheme thatmaximizes memory efficiency in terms of learning accuracy and capacityto enable both fast and slow stable learning of sensory features.

Benefits and Applications

Benefits of these methods and systems include providing a single processfor identifying, learning, localizing, and tracking objects in visualscenes provided by cameras. Exemplary methods allow quick and stablelearning of new patterns without the need to retrain the system, whilereducing network (system) size and communication between systemcomponents with respect to competing models. The method allowscontinuous learning of arbitrary sensory representations in hierarchiesof rate-based or spike-based neural processing stages connected byadaptive (learnable) synaptic weights. An additional benefit of thismethod is to allow fast learning of new stimuli without the need tointerrupt the functioning of the machine, e.g., allowing a robot with acamera to quickly learn the identity of a new, previously unlearnedinput without the need to retrain previously seen input.

The novel method presented herein can have application in designingsoftware to either extract information or control mobile robots orcameras. In particular, the method allows these machines to increasetheir knowledge base over time without the need to retrain the system onthe entire knowledge base.

OpenEye Overview

OpenEye is an artificial visual system operating on visual data. TheOpenEye model is comprised of four main modules: the Environment Module,the Where system, the What system, and an external module that canprovide a teaching signal to the what system (FIG. 1). These fourcomponents will be discussed in detail below.

The Environment Module (100) abstracts interactions between the visionsystem and the environment, which can be a virtual environment or a realenvironment sampled by a fix/pan-tilt camera, a robot-mounted camera, orother visual (or non-visual) sensory system. This module delivers avisual image to the visual system and executes camera movement commands,which emulate human eye movements. The environment module allows OpenEyeto interact with the environment: virtual or real, static or dynamic,real time or prerecorded.

One task of the Where System (130) is to decide where the sensory systemshould look based on salient image properties extracted from the visualimage, or based on information coming from the What System pertaining tothe identity of objects in the environments, and/or the scene identityas a whole. Processing of a visual image by the where system moduleincludes aspects of the mammalian lateral geniculate nucleus (LGN),primary visual cortex (V1), and higher cortices (V2, MT, MST)processing. The image obtained from the environment module in retinalcoordinates undergoes a log-polar transformation to simulatespace-variant sampling of the visual input and extraction of featuressuch as (but not limited to) edge, contour, color, and luminance.OpenEye's functioning is not limited to log-polar sampling, and canoperate with other space-variant transformations, such as thereciprocal-wedge transform (Tong and Li, 1995), or the pyramid method(Adelson et. Al, 1984), as examples.

Also known as the dorsal stream in vision literature (Mishkin andUngerleider 1982; Webster et al., 1994), OpenEye's Where Systemgenerates camera movements in order sample an image by foveation on thespatial location it selects as the most salient, where saliency can bedetermined by sensory input or semantic (What System) information.Foveation is achieved by centering the sensor in the object of interest,so that the object is likely to fall in the center of the space-variantrepresentation. A form-fitting attentional shroud (namely a signal thatfits the form, or shape, of an object, similarly to the way a shroud orveil fits the surface it rests on) is then formed around the foveatedobject. The shroud serves to suppress surrounding objects in order toisolate the object of interest for learning in the What System, andenables the system to trigger further camera movements centeredexclusively on this enshrouded object. The ability of the Where Systemto form this attentional shroud around a single object has the addedbenefit of detecting when a foveation has left the previous object ofinterest. This change in foveated object produces a reset signal thatrepresents temporal discontinuity between the foveations and is used bythe What System to regulate learning, with the result of allowingOpenEye to group multiple views of an object (but not other objects, orthe background) into coherent object categories. Another function of theWhere System is to maintain a visual working memory of previouslyfoveated locations such that the camera does not persistently choose thesame point of fixation. Together with the Environment Module, the WhereSystem forms the Where Pathway (140) that concerns with spatialinteraction with the environment and spatial processing.

The What System (150) includes a hierarchy of classifiers thatcollectively learn to visually recognize an arbitrary number of objectsregardless of each object's position and orientation relative to thesensor(s), e.g. a camera. The What System receives an object's featurerepresentation as input from the Where System. Views are then clusteredin an incremental, unsupervised fashion into object representationsbased either on their similarity or according to their temporalcontinuity as signaled by the Where System. The Where System provides ashroud-based reset signal, discussed later, that informs the What Systemwhen seemingly different views are part of the same or different object;this signal is important to OpenEye's ability to learn pose-invariantobject representations (Fazl et al., 2009). An optional external Teacher(160) provides a supervised learning environment that not only improvesclassification accuracy and learning speed but also dynamically createsa user-friendly search interface to the visual system's learnedknowledge. Because of the hierarchical separation of unsupervised viewlearning and supervised object-label learning, the What System can beswitched between unsupervised and supervised learning modes at any time.

The What system and Teacher together form the What Pathway (170),modeled upon the ventral visual processing stream in the mammalianbrain, which concerns the identity of those objects viewed by OpenEye.FIG. 1 depicts the overall structure of OpenEye. Each module isdescribed below with its corresponding block number.

Encoding OpenEye Activity

A critical task for OpenEye operation is switching between thecoordinate systems centered on the on the robot/camera/sensor(ego-centric), the environment (image-centric or world-centric), andbetween metrics systems (e.g., Cartesian or log-polar). For example, theimage is sampled using retinal (log-polar) metric, or other (e.g.,pyramid or reciprocal-wedge), but the signal for the cameral to move andhow much to adjust the pitch, yaw is provided in Cartesian (linear)metric. One role of the Where System concerns translating betweenrepresentations of a signal to different coordinate bases. For clarity,each coordinate system is defined with a term that refers to where thesystem is centered followed by a term that defines the distance metricof the reference frame. Reference frames can be centered at threepossible locations: 1) sensor-centered, 2) ego-centered, and 3)image-centered. Sensor-centered refers to a coordinate system where the(0, 0) location resides at the position of the current camera center.Ego-centered refers to a coordinate system where (0, 0) corresponds to aneutral position of a sensor, with respect which the camera center maybe shifted or rotated. This robot-centered coordinate system caninterface with other software systems to provide object location datarelative to the physical system or, when paired with global navigationdata, to provide a global object location. Image-centered refers to areference frame in which the (0, 0) location is at the image center.Image-centered can also be interpreted as global coordinates orscene-centered when the scene is dynamically changing. Correspondinglythere are at least three set of dimensions used in OpenEye: ImageDimensions [W_(i) H_(i)], Sensor Movement Range [W_(e) H_(e)], andSensor Dimensions [W_(s) H_(s)] that represent log-polar transform ofthe Sensor Movement Range. This notation is used in OpenEye descriptionbelow.

There are at least two distance metrics in the coordinate frames: 1)log-polar, and 2) Cartesian. The log-polar distance metric reflects howthe eye naturally samples the image and image representation in primaryvisual cortex, and is employed in the described system by performing aspace-variant (log-polar in this case, but other methods could be used)transformation to the ray input, while the Cartesian distance metric ismore pertinent when mapping representations onto the real word or forinvoking linear control of the eye/camera. In the figures and textbelow, coordinate frame are referred to as a combination of where it iscentered and what defines its distance.

FIGS. 1-3 depicts aspects of the What and Where systems of an exampleOpenEye system. FIG. 1 shows the Environment Module (120) and the WhereSystem (130), which collectively constitute the Where Pathway (140). Theenvironment module 120 includes an RGB image sensor 100, which mayacquire still and/or video images, whose field of view can be shifted,narrowed, and/or expanded with one or more actuators 110, including butnot limited to zoom lenses, tip/tilt stages, translation stages, etc.The environment module 120 provides both image data from the imagesensor 100 and actuation data (sensor position data) from theactuator(s) 110 to the Where system 130, which in turn providesprocessed image data to the What system 150. The environment module 120also provides actuation data (sensor position data) from the actuator(s)110 to the Teacher 160, which forms part of the What pathway 170 withthe What system 150.

FIG. 2 shows the Where system 130 in greater detail. A first log-polartransformation block 260 in the Where system 130 performs a log-polartransformation on the image data from the image sensor 100 as describedin greater detail below. A feature extraction block 240 identifiesfeatures in the transformed image data, which is segmented into boundedregions by a segmentation block 180. A figure/segregation block 210segregates the bounded regions to form a spatial shroud that fits thefoveated region of the image. The figure/segregation block 210 providesa representation of this spatial shroud to the What system 150.

FIG. 2 also shows that the actuator(s) 100 provide sensor position datato a foveation memory 250 and an inhibition of return block 220, whichtogether prevent the image sensor from foveating the same portions ofthe scene (acquiring and/or processing imagery of the same portions ofthe scene, e.g., at enhanced resolution) unnecessarily. A secondlog-polar transformation block 230 performs a log-polar transformationon the output of the inhibition of return block and passes thetransformed output to a hot spot selection block 190, which determinesthe next portion of the scene for foveation. A reverse log-polartransformation block 270 transforms the output vector into the frame ofreference used by the actuator(s) 100 and provides the transformedoutput vector to the actuator(s) 100 for actuation of the sensor 100. Atemporal object continuity block 200 processes another copy of the hotspot selection block output to determine if the next foveation locationfalls off the current object surface. If so, the temporal objectcontinuity block 200 transmits a “reset” signal to the What system 150

FIG. 3 shows the What system 150 in greater detail. The What system 150uses data from the temporal object continuity block 200, the featureextraction block 240, and the figure/ground segregation block 210 toidentify and locate objects in the scene imaged by the image sensor 100.A view layer 280 uses features and shroud data from the Where system 130to cluster shroud-gated visual representations of object views accordingto their feature similarity. A disambiguation map block 310 generates adisambiguation map of the scene based on these representations from theview layer 280.

The object layer 290 uses the representations from the view layer 280 tolearn pose-invariant object representations by associating differentview prototypes from the view layer 280 according to their temporalcontinuity provided by the reset signal from the Where system 130. Thisyields an identity confidence measure, which can be fed into a namelayer 300 that groups different objects under the same user label, whichmay be obtained from an optional teacher 160. The optional teacher 160shapes the association between objects and their labels and feeds thisinformation from the Name layer 300 to the Object layer 290 and Viewlayer 280 to the speed and accuracy of future object learning.

The What system and the Where system can be implemented in hardware,firmware, software, or a suitable combination thereof. For example, theWhat and Where systems may be implemented as processor-implementableinstructions that are stored in non-transient form in one or morememories located in or on a robot, such as a unmanned aerial, ground, orsubmersible vehicle. Some or all of the processor-implementableinstructions may also be stored on remote memory, such memory in oraccessible by a server that communicates with the robot via a wirelesscommunication link (e.g., a radio-frequency or optical link).

The robot may include one or more processors that are coupled to thememory and configured to execute the instructions so as to implement theWhat and Where systems, including the individual modules shown in FIGS.1-4. For example, the robot may execute the instructions with a centralprocessing unit (CPU) and a graphics processing unit (GPU), e.g., asdisclosed in U.S. Pat. No. 8,648,867, which is incorporated herein byreference in its entirety. The processor(s) can also be implemented asapplication specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), and/or other device or component as understood inthe art.

In some embodiments, some or all of the processors may be locatedremotely—that is, not on or in the robot. For example, the processors(include GPUs) may be located in one or more smart phones, tablets,and/or single board computers (SBCs). The processors may also form partor all of a cluster computing environment, with each processor in thecluster dedicated to particular task or group of tasks. In theseembodiments, the processors may communicate with sensors, actuators, andother devices and components on or in the robot via a suitablecommunications link, such as a radio-frequency or optical communicationslink.

FIG. 5 illustrates an OpenEye system 500 used to control a wheeled robot510. The OpenEye system 500 includes a computing device 504, such as atablet computer or other electronic device with wireless capabilities,that is controlled by a user 502. The computing device 504 communicateswith the robot 510, which includes an image sensor 512 and an antenna514, via a wireless link. The user 502 issues commands to the robot 510via software running on the computing device 504, a processor (notshown) on the robot 510, and/or on other cloud-based processors (notshown).

In operation, the image sensor 512 can be oriented and/or positionedeither by the user when manually operating the robot or automatically bythe software. For example, the image sensor 512 may be mounted on apan/tilt stage, translation stage, or rotation stage that can beactuated to change the image sensor's orientation and/or position. Theimage sensor 512 may also have a (motorized) zoom lens that can be usedto zoom in or out on certain portions of the environment. In addition,or instead, the image sensor 512 can be oriented or positioned asdesired by moving the robot 510. In some cases, the image sensor 512 maystatic with respect to the robot 510; this is roughly equivalent tosomebody without, say, neck and eye muscles. In order to change thestatic image sensor's point of view, the body of the robot rotatesand/or moves, e.g., using wheels or legs for ground robots, propellersfor drones, thrusters for submersible robots, etc.

Environment Module (120)

This Environment Module abstracts away the source of visual imagery(cameras, real or virtual, or other sensors, e.g. LIDAR) and appliessensor movement commands in the manner consistent with the environmentin which OpenEye currently operates. OpenEye supports the followingenvironments:

-   -   Static Scenes—JPEG, PNG images, etc.    -   Dynamic Scenes—movie files (.avi, .mp4, etc.)    -   Camera—Real 3d visual world    -   Virtual Camera—virtual environment, based on the JMonkey game        engine

Concrete implementations of this module are specific to the environment,but the input and the output should comply with the specification below.

RGB Sensor (100)

RGB delivers the RGB image sampled from the environment as directed bythe RGB Sensor Actuator. The later simulates eye movement by moving thecamera.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Sensor Vector 0 Unsigned, 2 Cartesian Ego- (e.g., camera) Bytecentered Position* *This input does not have to be used in allenvironments. If the environment includes Sensor Movement Actuator (forexample, P&T camera, or Virtual Camera), this input is not necessarilyused by the RGB Sensor.

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase RGB Sensor Vector [W_(s), H_(s)] Unsigned, 3 Cartesian Sensor-Field Byte centered

Sensor Movement Actuator (110)

Sensor Movement Actuator implements sensor (e.g., camera) movementcommands if they are supported by the environment, otherwise this modulereturns eye position in ego-centric coordinates.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Sensor (Eye) Vector 0 Unsigned, 2 Cartesian Ego- Position Bytecentered Location to Vector 0 Unsigned, 2 Cartesian Sensor- foveate Bytecentered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Sensor (Eye) Vector 0 Unsigned, 2 Cartesian Ego- Position Bytecentered

Where System (130)

One function of the Where System is to produce a foveated view of theobject to be interpreted by the What System, to select the next locationto foveate based on sensory and internal semantic information, and todetermine and track the position of objects in the visual field andreturn their coordinates. The diagram of the Where System is presentedon FIG. 2. All modules part of the Where System are enclosed in themodule described in (130). The Where System receives the video imagefrom the environment module and produces camera movement commands to beexecuted by the environment module (120). The Where System supplies theWhat System with the view of the object it currently looks at and theReset signal, which marks the beginning of the object foveationsequence. The detailed description of the Where System modules ispresented below.

Log-Polar Transformations

Several modules (230, 260, 270) perform transformation between log-polarand Cartesian encoding of the distance metric. OpenEye adheres to abio-inspired log-polar transform of the input image, but the model canbe used with different transform. The log-polar transform is applied tothe RGB sensor subtending 136 degrees of visual angle, close to thatreported in humans and other animals (Traver and Bernardino, 2010). Thelog-polar metric in space encoding is used across both OpenEye Where andWhat Systems and transformed back to the Cartesian metric by (230) toreference the external world beyond the current view, which is requiredby the Environment Module (120), the Foveation Memory module (250), andthe Inhibition of Return module (220) in the Where Pathway. AllLog-polar transformation modules share the parameters that specifydimensions of log-polar [w_(s) h_(s)] and Cartesian image [W_(s) H_(s)].

Log-Polar Transformation of Retinal RGB Image (260)

The image sampled at the foveated location undergoes log-polartransformation that amounts to space-variant sampling with higherresolution in the foveal area and much coarser resolution that fallswith eccentricity outside the foveal region (Traver and Bernardino,2010). This provides some invariance to translation/rotation and to savecomputational bandwidth while at the same time to acquire details at thelocation of the image that present the highest interest and is the mosteffective for the image representation.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase RGB Image I Vector [W_(s), H_(s)] Unsigned, 3 Cartesian Sensor-Field Byte centered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Log-Polar Vector [w_(s), h_(s)] Unsigned, 3 Cartesian Sensor- RGBImage Field Byte centered O

Processing:

$O_{ij} = \left\{ {\begin{matrix}{{{I_{ij}\mspace{14mu} {d\left( {i,j} \right)}} < {r_{fov}\mspace{14mu} {d\left( {i,j} \right)}}} = \sqrt{\left( {i - i_{0}} \right)^{2} + \left( {j - j_{0}} \right)^{2}}} \\{{{I_{xy}\mspace{14mu} {d\left( {i,j} \right)}} \geq {r_{fov}\mspace{14mu} {d\left( {x,y} \right)}}} = {\ln \mspace{14mu} {d\left( {i,j} \right)}}}\end{matrix};{{{Where}\text{:}r_{fov}} = \frac{\max \left( {W_{s},W_{h}} \right)}{24}};{i_{0} = \frac{\left( {W_{h} - 1} \right)}{2}};{j_{0} = {\frac{\left( {W_{s} - 1} \right)}{2}.}}} \right.$

Log-Polar Transformation of Inhibition of Return (230)

Similarly to retinal image, inhibition of return undergoes log-polartransformation in order to prevent the HotSpot Selection Module (190)from repeated foveations.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Inhibition of Scalar [W_(s), H_(s)] Unsigned, 0 Cartesian Sensor-Return I Field Byte centered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Log-Polar Scalar [w_(s), h_(s)] Unsigned, 0 Cartesian Sensor-inhibition Field Byte centered of return O

Processing:

$O_{ij} = \left\{ {\begin{matrix}{{{{sign}\mspace{14mu} I_{ij}\mspace{14mu} {d\left( {i,j} \right)}} < {r_{fov}\mspace{14mu} {d\left( {i,j} \right)}}} = \sqrt{\left( {i - i_{0}} \right)^{2} + \left( {j - j_{0}} \right)^{2}}} \\{{{{sign}\mspace{14mu} I_{xy}\mspace{14mu} {d\left( {i,j} \right)}} \geq {r_{fov}\mspace{14mu} {d\left( {x,y} \right)}}} = {\ln \mspace{14mu} {d\left( {i,j} \right)}}}\end{matrix};{{{Where}\text{:}r_{fov}} = \frac{\max \left( {W_{s},W_{h}} \right)}{24}};{i_{0} = \frac{\left( {W_{h} - 1} \right)}{2}};{j = {\frac{\left( {W_{s} - 1} \right)}{2}.}}} \right.$

Reverse Log-Polar Transformation of Inhibition of Return (270)

HotSpot selected in the Log-polar view in sensor-centered coordinates(190) needs to be transformed back to Cartesian metric by (230) beforeit can be converted into sensor movement command by the EnvironmentSensor Movement Actuator (110). Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Hot Spot Vector 0 Unsigned, 2 Log-polar Sensor- I = [i j] Bytecentered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Hot Spot vector 0 Unsigned, 2 Cartesian Sensor- O = [x y] Bytecentered

Processing:

$O = \left\{ {\begin{matrix}{{{\left\lbrack {i\mspace{11mu} j} \right\rbrack \mspace{14mu} {d\left( {i,j} \right)}} < {r_{fov}\mspace{14mu} {d\left( {i,j} \right)}}} = \sqrt{\left( {i - i_{0}} \right)^{2} + \left( {j - j_{0}} \right)^{2}}} \\{{{\lbrack{xy}\rbrack \; {d\left( {i,j} \right)}} \geq {r_{fov}\mspace{14mu} {d\left( {i,j} \right)}}} = {\ln \mspace{14mu} {d\left( {x,y} \right)}}}\end{matrix};{{{Where}\text{:}r_{fov}} = \frac{\max \left( {W_{s},W_{h}} \right)}{24}};{i_{0} = \frac{H_{s} - 1}{2}};{j_{0} = {\frac{W_{s} - 1}{2}.}}} \right.$

Feature Extraction (240)

Feature Extraction (240) includes, but is not limited to, computation ofluminance and color. Other features could include motion, or SIFTfeatures (Lowe, 2004). “Features” can be:

1. A property of an image that can be associated with each imagelocation;2. A scalar (luminance, 0-dimensions) or vector (color, 1 dimension);3. A numerical (integer, or real, e.g. luminance, color) or binary(Boolean, e.g., is an edge associated with this particular pixel) value.More abstract properties (e.g., “edgeness”) can also be represented by anumerical feature—strength of the edge.

The description below specifies the features currently implemented inOpenEye, but the description below should not be intended to limitOpenEye applicability to these features alone.

Luminance, Color

Luminance and Color are extracted from the Log-polar RGB Image.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase RGB Image Vector [w_(s) h_(s)] Unsigned, 3 Log-polar Sensor- I = [rg b] Field Byte centered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Color Vector [w_(s) h_(s)] Unsigned, 3 Log-polar Sensor- O_(c) = [hs v] Field Byte centered Luminance Scalar [w_(s) h_(s)] Unsigned, 0Log-polar Sensor- O_(L) Field Byte centered

Processing

O_(l) = r + g + b O_(c) = [h  s  v] Where:${v = \underset{S}{\max \left( {r\mspace{11mu} g\mspace{11mu} b} \right)}};$${s = \frac{v - {\min \left( {r\mspace{11mu} g\mspace{11mu} b} \right)}}{v}};$$h = \left\{ {\begin{matrix}\frac{60\left( {g - b} \right)}{v - {\min \mspace{11mu} \left( {r\mspace{11mu} g\mspace{11mu} b} \right)}} & {{{if}\mspace{14mu} v} = r} \\\frac{120 + {60\left( {b - r} \right)}}{\min \mspace{11mu} \left( {r\mspace{11mu} g\mspace{11mu} b} \right)} & {{{if}\mspace{14mu} v} = g} \\\frac{240 + {60\left( {r - g} \right)}}{\min \mspace{11mu} \left( {r\mspace{11mu} g\mspace{11mu} b} \right)} & {{{if}\mspace{14mu} v} = b}\end{matrix}.} \right.$

Segmentation (180)

This module builds preliminary segmentation producing binary image thatrepresent closed (bounded) regions (Suzuki & Abe, 1985). This isachieved by using OpenCV function findContours, which operates on edgesproduced by the Canny edge detector (Canny, 1986). The result is theimage with pixels set to 1 at the locations that belong to the boundedregions.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Luminance Scalar [w_(s) h_(s)] Unsigned, 0 Log-polar Sensor- Bytecentered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Contours Scalar [w_(s) h_(s)] Binary, 0 Log-polar Sensor- FieldByte centered

Figure/Ground Segregation (180)

This module builds a shroud around the object at the center of the view.This is achieved via a seeded flood fill algorithm, which uses theOpenCV floodFill function. This algorithm fills a connected componentstarting from the center of the log-polar image produced by thesegmentation module (180). Connectivity is determined by the brightnesscloseness of the neighbor pixels. As the result this step produces ashroud (Fazl et al., 2009), roughly fitting the form of the closedregion that includes the foveated location (the center of the image).

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Contours I Scalar [w_(s) h_(s)] Binary, 0 Log-polar Sensor- FieldByte centered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Shroud O Scalar [w_(s) h_(s)] Binary, 0 Log-polar Sensor- FieldByte centered

Processing:

$O_{ij} = \left\{ {\begin{matrix}I_{ij} & {{\sum\limits_{\substack{{i = {- 1}},0,1 \\ {j = {- 1}},0,1}}\left( {I_{ij}==I_{{i + n},{j + m}}} \right)} > 1} \\0 & {{\sum\limits_{\substack{{i = {- 1}},0,1 \\ {j = {- 1}},0,1}}\left( {I_{ij}==I_{{i + n},{j + m}}} \right)} = 1}\end{matrix};} \right.$

The Figure/ground segregation module (180) can also be extended toaccept input from the What System, for instance in the form of semanticinformation pertaining the identity of pixels which can be obtained viaa fast processing of the visual information that bypasses the WhereSystem. For instance, a separate What System can be trained torecognize, on a pixel-by-pixel basis, areas in the image. E.g., theseparate What System can initially classify areas of the image as “sky”,“grass”, “road”, and this information can be used as input to theFigure/ground segregation module (180) as additional input to drivefigure/ground segregation.

Hot Spot Selection (190)

This module produces a vector that determines the next foveationlocation. The module determines the most salient locations on the imageby using the OpenCV function goodFeaturesToTrack, which finds the mostprominent corners in the image as described in (Shi and Tomasi, 1994).The image passed to the corner finding algorithm is the luminancefeature produced by the feature extraction module (240). The Inhibitionof Return signal produced by the log-polar transformation module (230)disables the non-zero locations on the image to be selected as the nextfoveation position.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Contours I Scalar [w_(s) h_(s)] Unsigned, 0 Log-polar Sensor- FieldByte centered Inhibition of Scalar [w_(s) h_(s)] Binary, 0 Log-polarSensor- Return mask Field Byte centered I_(r)

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase HotSpot O Vector 0 Unsigned 2 Log-polar Sensor- Byte centered

Processing:

Finding most prominent corners in the input image follows the algorithmbelow:

1. Covariation matrix M_(ij) is computed for every pixel in the imagefor the 3×3 neighborhood N_(ij)

$M = {\begin{bmatrix}{\sum\left( \frac{dI}{dx} \right)^{2}} & {\sum\left( {\frac{dI}{dx}\frac{dI}{dy}} \right)^{2}} \\N_{ij} & N_{ij} \\{\sum\left( {\frac{dI}{dx}\frac{dI}{dy}} \right)^{2}} & {\sum\left( \frac{dI}{dx} \right)^{2}} \\N_{ij} & N_{ij}\end{bmatrix}.}$

The derivatives are computed using the Sobel operator by convolving theinput with the following kernels

$\frac{dl}{dx} = {{conv}\left( {I,\begin{bmatrix}{- 1} & 0 & {- 1} \\{- 2} & 0 & 2 \\{- 1} & 0 & {- 1}\end{bmatrix}} \right)}$ and$\frac{dl}{dy} = {{conv}\left( {I,\begin{bmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\{- 1} & {- 2} & {- 1}\end{bmatrix}} \right)}$ and λ = min (λ₁, λ₂),

where λ₁,λ₂ are unsorted eigenvalues of M2. Eigen values are used as a measure of corner quality, which aresorted and the locations below the quality threshold T_(q)(T_(q).=0.001) are rejected.

At this point the list of candidates O_(c) containing locations in theimage I where eigenvalues exceed the threshold T_(q) is produced. Themask signal I_(r) is used to exclude the candidates, for which I_(r)≠0.

3. The remaining candidates are tested in descending order as newseeding points, with the floodFill function (see 3.4). If figureground/segregation is possible for the given candidate (the seeded valuedoes not flood over log-polar image boundary, i.e. image corners) itbecomes the next selected foveation point, in other words—the outputvector O. Otherwise the process is repeated for the next candidate untilgood hot spot is found capable of producing a shroud.

Temporal Object Continuity (200)

In order to build view invariant object identity, OpenEye may maintaintemporal continuity between subsequent object foveations. OpenEyedetermines if the next foveation location falls off the current objectsurface in order to signal the object recognition system that buildingof the new object identity begins or continues. This is achieved viaproducing the RESET signal, which is set to 1 in the next cycle when theselected new foveation location falls off the shroud (output of module210) built from seeded activity in the center point of the view.

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Shroud I Scalar [w_(s) h_(s)] Binary, 0 Log-polar Sensor- FieldByte centered HotSpot Vector 0 Unsigned, 2 Log-polar Sensor- I_(h) = [xy] Byte centered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase RESET O Scalar 0 Binary 0 Log-polar Sensor- Byte centered

Processing:

O=I _(xy);

Implementation of temporal continuity requires referencing discoveredobjects in space. Since individual components of the OpenEye requiredifferent encoding of space to refer to the object's location, severalOpenEye computations involve translating spatial references betweencoordinate frames, different metrics, and different types of coordinatesystem. The following table summarizes usage of spatial representationsby individual OpenEye Components:

Frame of Open Eye Data Reference Metrics Coordinate type Object MapCamera-centered Linear Spherical (pitch, yaw, distance) Object MemoryAllocentric Linear Cartesian (X, Y, Z) Camera Position Egocentric LinearEuler (pitch, yaw, Orientation roll = 0) Robot Location AllocentricLinear Cartesian (X, Y, Z) Robot Orientation Allocentric Linear Euler(yaw, pitch, and roll) Shroud Camera-centered Log Cartesian (X, Y)Inhibition of Return Camera-centered Log Cartesian (X, Y) DisambiguationMap Camera-centered Log Cartesian (X, Y) Hot Spot Camera-centered LogCartesian (X, Y) (Next Foveation)

In an allocentric frame of reference, the location of one object isdefined relative to the location of other objects. (In an egocentricframe of reference, on the other hand, an object's location is definedrelative to the body axes of the robot.) For example, an allocentriccoordinate frame can be aligned as follows: Axis Y—up vertically, axis Zwith initial robot heading, and axis X in the direction perpendicular toaxes Y and Z. In OpenCV (the software framework used to develop OpenEye)the direction of axis y is reversed. OpenCV representation is used forcamera-centered frame of reference. Orientation is encoded by the Eulerangles: yaw, pitch, and roll. Camera-centered representations could beexpressed in spherical coordinates, with the X, Y treated as yaw andpitch angles, while the distance to the projected pixel will correspondto the radius R. This spherical interpretation of the camera-centeredprojection will be used to describe coordinate transformation. Two typesof coordinate transformation are crucial for the approach to temporalcontinuity:

1) Translation of spatial reference from camera-centered to allocentriccoordinates; and2) Mapping location in allocentric coordinates to the camera-centeredframe of reference.

The first may be utilized for learning discovered objects' locations;the second may be utilized for maintaining awareness of discoveredobjects in the field of view. After log-polar transformation, OpenEyesegments the image and shroud a foveated object. Once the object shroudis produced, the location of the foveated object can be added to theobject memory, where it is stored in allocentric coordinates for futurereferencing. Thus, position of the object in the sensed image should betransformed to the allocentric location given the current robotposition, orientation as well as camera orientation. Finally a newhotspot is selected, which should cause new saccade, sensing new imageand translating the hotspot position into a frame, where a new shroudwill be produced using seeded filling-in from the adjusted hotspotposition, which should take into account robot ego-motion.

FIG. 6 provides an overview of how the OpenEye determines temporalobject continuity. In block 602, an image sensor, which may or may notbe mounted to the robot, obtains imagery of the robot's environment. Oneor more OpenEye processors translate one or more these images from thecamera frame of reference to an allocentric frame of reference (e.g., alog-polar frame of reference) in block 604. The OpenEye processor thensegments the translated images in block 606. Next, the OpenEye processorconstructs a spatial shroud for a first image (block 608) based on thecurrent position and orientation of the input sensor and uses the shroudto identify an object in the first image (block 610). It thentranslates, rotates, skews, and/or otherwise transforms the shroud toaccount for the sensor's change in orientation and/or position betweenacquisition of the first image and a second image (block 612).

The processor then determines if the transformed shroud maps to anobject in the second image (block 614). If so, the processor determinesthat the object in the second image is the same as the object thatappears in the first image and learns the object's location (e.g.,stores a representation of the object, its features, and/or its positionin memory for later retrieval). At this point, the processor may use anactuator to orient and/or position the sensor in order to image adifferent portion of the robot's environment. If the shroud does notoverlap with an object sufficiently in the second image, the processordetermines that the objects are different and updates its memoryaccordingly. The processor may then actuate the sensor to obtainadditional images of the object and the surrounding portion of therobot's environment.

FIGS. 7A-7C illustrate the shroud construction and translation process.In FIG. 7A, the sensor is centered on a face 702, where the center ismarked by the dashed lines through the field of view. The OpenEyeprocessor 704 shroud is built around this face 702, shown by the grayshading in the diagram. After the sensor is reoriented and another imageacquired, the shroud 704 is translated and rotated to compensate for thesensor motion. If the sensor is now centered on a location marked by theshroud 704 in FIG. 7B, the system identifies that this object is thesame as the one previously viewed. If the sensor is instead centered ona location off of the shroud 704, as in FIG. 7C, the system identifiesand learns views of a new object.

Note that the What system (aka the classifier or semantics module) canalso contribute to controlling the Where system (aka the spatialattention module). In particular, if the What system has gathered enoughevidence (namely, a certain number of classifications where confidenceis high) about the foveated object, it may cause the Where system tostop foveating that object, producing Inhibition Of Return (IOR) for afew time steps in the future, so as to bias the visual system toclassify other objects in the scene.

Translating Spatial References from Camera-Centered to AllocentricCoordinates

Location in allocentric coordinates can be computed from local referenceas following:

$\begin{matrix}{{\begin{matrix}X_{o} \\Y_{o} \\Z_{o}\end{matrix}} = {{{R_{A}}^{- 1} \times {\begin{matrix}x_{o} \\y_{o} \\z_{o}\end{matrix}}} + T_{A}}} & \left( {3.6{.1}} \right)\end{matrix}$

Where:

$\quad{\begin{matrix}X_{o} \\Y_{o} \\Z_{o}\end{matrix}}$

Position vector in the allocentric frame of reference.

|R_(A)| robot rotation matrix in allocentric coordinates

$\quad{\begin{matrix}x_{o} \\y_{o} \\z_{o}\end{matrix}}$

Position vector in the egocentric frame of reference.

T_(A) is the robot translation vector in egocentric coordinates, whichcan be obtained directly from the GPS system in virtual environment.

|R_(A)| can be computed from GPS orientation sensory signal asfollowing:

$\begin{matrix}{R_{A} = {\begin{matrix}{{\cos (y)}{\cos (p)}} & {{{\sin (y)}{\sin (r)}} - {{\cos (y)}{\sin (p)}{\cos (r)}}} & {{{\sin (y)}{\cos (r)}} + {{\cos (y)}{\sin (p)}{\sin (r)}}} \\{\sin (p)} & {{\cos (p)}{\cos (r)}} & {{- {\cos (p)}}{\sin (r)}} \\{{- {\sin (y)}}{\cos (p)}} & {{{\sin (y)}{\sin (p)}{\cos (r)}} + {{\cos (y)}{\sin (r)}}} & {{{- {\sin (y)}}{\sin (p)}{\sin (r)}} + {{\cos (y)}{\cos (r)}}}\end{matrix}}} & \left( {3.6{.2}} \right)\end{matrix}$

Where y, p, and r correspond to yaw, pitch, and roll of the robot inallocentric coordinates.

$\quad{\begin{matrix}x_{o} \\y_{o} \\z_{o}\end{matrix}}$

is computed from the position (x_(s)y_(s)) in camera-centeredcoordinates, which is the output of the hot spot selection module.First, (x_(s)y_(s)) should be translated to egocentric frame usingcamera position (E_(x),E_(y)) in egocentric frame of reference. Thisenables us to compute location direction in the egocentric frame ofreference. The angles in the egocentric coordinate system can becomputed according to (3.6.3).

X _(e) =x _(s) +E _(x);  (3.6.3a)

Y _(s) =x _(y) +E _(y)  (3.6.3b)

Now we can compute the location's Cartesian egocentric coordinates forthe equation (3.6.1) using the distance information obtained from thedepth sensor. Taking into account reversed orientation of the y axis inOpenCV, position vector

$\quad{\begin{matrix}x_{o} \\y_{o} \\z_{o}\end{matrix}}$

can be computed according to (3.6.4)

$\begin{matrix}{{\begin{matrix}x_{o} \\y_{o} \\z_{o}\end{matrix}} = {{\begin{matrix}{{- {\cos \left( X_{e} \right)}}{\sin \left( Y_{e} \right)}} \\{{- {\sin \left( X_{e} \right)}}{\sin \left( Y_{e} \right)}} \\{\cos \left( Y_{e} \right)}\end{matrix}} \times d_{xy}}} & (3.64)\end{matrix}$

Where:

d_(xy) is the depth signal available from the depth sensor in virtualenvironment. d_(xy) corresponds to the distance from the robot to thelocation represented by pixel c The range of distances sensed by thedepth sensor in virtual environment is determined by the frustumsettings for the virtual camera and currently is set to the intervalfrom 0.5 meters to 1000 meters. The distances are normalized between 0and 1. Everything closer the low frustum bound is sensed as 0.Everything further than upper frustum bound is sensed as 1.

Computing hotspot location requires translation of position of theselected pixel from log to linear metric with subsequent translation ofthe linear pixel position into the spherical angles specific to cameraangle of view and the resolution.

x _(s) =P _(x) *R _(x),  (3.6.5a)

y _(s) =P _(y) *R _(y)  (3.6.5b)

Where:

(P_(x) P_(y)) is the position of the selected hotspot translated fromlog to linear metric(R_(x),R_(y)) is the resolution in radians/pixel in x and y directioncorrespondingly.R_(x) and R_(y) can be determined from the size of the image (S_(x),S_(y)) and the size of the field of view (V_(x), V_(y)), which for oursimulation was set to (600,600) and (π/2, π/2) correspondingly. P_(x)and P_(y) are computed using reverse log-polar mapping as following

$\begin{matrix}{\left( {P_{x}P_{y}} \right) = \left\{ {\begin{matrix}{{\left\lbrack {i\mspace{11mu} j} \right\rbrack {d\left( {i,j} \right)}} < R_{f}} & {{d\left( {i,j} \right)} = \sqrt{\left( {i - i_{0}} \right)^{2} + \left( {j - j_{0}} \right)^{2}}} \\{{\lbrack{xy}\rbrack {d\left( {i,j} \right)}} \geq R_{f}} & {{d\left( {i,j} \right)} = {\ln \; {d\left( {x,y} \right)}}}\end{matrix};} \right.} & \left( {3.6{.6}} \right)\end{matrix}$

Where:

[i j] is a hotspot location (row, column) in log-polar view in thecamera-centered frame of reference

${R_{f} = \frac{\max \left( {W_{s},H_{s}} \right)}{24}};{i_{0} = \frac{H_{s} - 1}{2}};{j_{0} = \frac{W_{s} - 1}{2}};$${H_{s} = {2R_{f}{\log \left( {\frac{s_{y}}{2R_{f}} + 1} \right)}}};$$W_{s} = {2R_{f}{\log \left( {\frac{s_{x}}{2R_{f}} + 1} \right)}}$

R_(f) is foveal radius as percentage of entire retina.W_(s) and H_(s) correspond to log-polar mapping of the image sensordimensions (S_(x), S_(y))

When an object, whose shape is defined by the shroud is learned by theWhat system, its location and the shroud is stored in the Object Memoryin allocentric coordinates [X,Y,Z]. The allocentric coordinate system isaligned with the initial robot location and orientation so the Z axis isaligned with the heading of the robot.

Mapping Location from Allocentric to the Camera-centered Frame ofReference

Pixel location in camera-centered coordinates (x_(s),y_(s)) can becomputed from its egocentric Cartesian coordinates (x₀, y₀, z₀) asfollowing:

$\begin{matrix}{x_{s} = \frac{E_{x} + {\tan^{- 1}\frac{y_{0}}{x_{0}}}}{R_{x}}} & \left( {3.6{.7}a} \right) \\{y_{s} = \frac{E_{y} + {\cos^{- 1}\frac{z_{0}}{d}}}{R_{y}}} & \left( {3.6{.7}b} \right)\end{matrix}$

Where:

d=√{square root over (x₀ ²+y₀ ²+z₀ ²)} is the distance to the point(x_(s) y_(s)) in the egocentric coordinates.

R_(x), and R_(y) is the resolution in radians/pixel in x and y directioncorrespondingly.E_(x) and E_(y) are camera position in egocentric coordinates

$\begin{matrix}{{\begin{matrix}x_{o} \\y_{o} \\z_{o}\end{matrix}} = {{{R_{A}} \times {\begin{matrix}X_{o} \\Y_{o} \\Z_{o}\end{matrix}}} - T_{A}}} & (8)\end{matrix}$

Where:

$\quad{\begin{matrix}X_{o} \\Y_{o} \\Z_{o}\end{matrix}}$

Position vector in the allocentric frame of reference.

|R_(A)| robot rotation matrix in allocentric coordinates

T_(A) is the robot translation vector in egocentric coordinates, whichcan be obtained directly from the GPS system in virtual environment.

OpenEye Implementation of Temporal Continuity

In OpenEye, temporal continuity is based on the ability to learn thelocation of the object selected during the foveation (camera movement)cycle. The location is learned by translating pixel positioncorresponding to the object in the camera-centered coordinates intoobject location in allocentric coordinates. This is done using equations(3.6.1)-(3.6.6).

To ensure awareness of previously learned objects, their locations istranslated from allocentric coordinates stored in object memory intocamera-centered representation at each foveation cycle using equations(3.6.7) and (3.6.8).

Similarly to objects, hot spot pixel position is translated toallocentric coordinates using equations (3.6.1)-(3.6.5) In the nextfoveation cycle, the position of hotspot is recomputed using equations(3.6.6) and (3.6.7) forming the shroud around the foveated object byseeded filling-in starting from the hotspot selected at the previousfoveation cycle

Foveation Memory (250)

The term “foveation” adopted below is borrowed from the neuroscienceliterature, where foveation represents the location of eye fixation.Foveation memory in OpenEye represents past foveation activity over thevisual image. When OpenEye operates on static images, foveation meanssampling of the image, at a particular (foveated) location. Usually sizeof the sampled image is much smaller than the entire image(scene) size.When OpenEye operates in real 3D or virtual environment, foveation issampling of that environment as the result of real or virtual cameramovement. The visual memory is maintained over the spatial area thatdepends on the environment. It could amount to the entire image as inthe case of static scene environment, or over the region of space thatis currently in the view as in the case of movies or virtualenvironment. Foveation memory inhibits foveations at the locations thathave been foveated in the past. After making a camera movement, OpenEyesets foveation activity at the maximum value (255), this activity decayswith each foveation and eventually, when it decays to 0, the location isenabled for new foveations. The Foveation Memory is maintained in theimage-centered coordinate frame. However, the input (Sensor Position,150) is provided in ego-centered coordinates.

The history gets updated with each new foveation cycles. The decay isimplemented as a decrement by one with each foveation step. Initialvalue immediately after foveation is set to FMAX. This means that thesame location cannot be foveated at least the next FMAX cycles.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Sensor Vector 0 Unsigned, 2 Cartesian Ego- Position Byte centeredI_(s) = [x, y] Foveation Scalar [W_(i) H_(i)] Unsigned 0 CartesianImage- Memory I_(f) Field Byte centered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Foveation Scalar [W_(i) H_(i)] Unsigned 0 Cartesian Image- Memory 0Field Byte centered

Processing:

O=I _(f)+255*K(x,y)−1,

where, K(x,y) is a Gaussian kernel centered at location [x, y], whicheffectively inhibits location around [x,y] by adding them to thefoveation memory

${K\left( {x,y} \right)} = \left\{ {\begin{matrix}{\frac{1}{2{\sigma}^{2}}e^{\frac{- 1}{2\sigma^{2}}d^{2}}} & {{d < {D\; {d\left( {i,j} \right)}}} = \sqrt{\left( {i - \frac{D}{2}} \right)^{2} + \left( {j - \frac{D}{2}} \right)^{2}}} \\0 & {otherwise}\end{matrix};\mspace{20mu} {\sigma = 4};{D = 1.}} \right.$

Inhibition of Return (220)

The purpose of the Inhibition of Return module (220) is to preventrepeated foveations at the same spatial location. To achieve that thismodule extracts the section of the foveation history around the nextfoveation location that falls in the view of the next saccade.

Input:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Sensor Position Vector 0 Unsigned, 2 Cartesian Ego- I_(s) = [x, y]Byte centered Foveation Scalar [W_(I) H_(I)] Unsigned 0 Cartesian Image-Memory I_(f) Field Byte centered

Output:

Data Data Element Element Name Type Dimensions Type Dimensions MetricBase Inhibition of Scalar [W_(s) H_(s)] Unsigned 0 Cartesian Sensor-return O Field Byte centered

Processing:

$\left\{ {\begin{matrix}{{{{I_{f}\left( {X,Y} \right)}0} < X < W_{I}};{0 < Y < H_{I}}} \\\left. {{0X} > 0} \middle| {X > W_{I}} \middle| {Y < 0} \middle| {Y > H_{I}} \right.\end{matrix},{{{{where}Y} = {y + i - i_{0}}};{X = {x + j - j_{0}}};{i_{0} = \frac{\left( {H_{s} - 1} \right)}{2}};{j_{0} = {\frac{\left( {W_{s} - 1} \right)}{2}.}}}} \right.$

What System (150)

The What System (150) learns the identities of objects and visualscenes. The What System may, for example, group object views and learnthem as a unified entity; maintain a lifelong memory while preventingmemory degradation and saturation; make inferences about views acquiredfrom the Where System, which objects they belong to, and the namesassociated with those objects; bias camera movements in the Where Systemin order to intelligently guide image sampling; and provide an interfaceby which an external user can communicate with and leverage the system'sknowledge.

The What System is implemented as a series of hierarchically organizedclassifiers that perform unsupervised view clustering, classification ofview categories into object categories based on the reset signal fromthe Where System, and supervised or unsupervised categorization ofobjects into name categories. After learning occurs, the activation of aname category primes the What system by inhibiting those objects andviews that are not associated with that name category, further tuningthe system by discouraging views from being shared between multipleobjects and names. The activation of a name category can come from abottom-up activation of a newly viewed object, persistent activationcaused by a previously viewed object, or through external activation byanother system or user. This external activation is provided by aTeacher (160) that represents the correct name of the foveated object toaid learning. OpenEye does not function in different modes to facilitatetraining or testing mechanisms, and it does not require a reset of thesystem upon transition to a new scene. FIG. 3 highlights the high levelsystem diagram of the What Pathway (170), which includes the What System(150) and the Teacher (160).

The inspiration for hierarchical clustering of views into objects andnames is detailed in the ARTScan model of visual learning (Fazl,Grossberg, and Mingolla, 2009). The Adaptive Resonance Theory (ART)learning scheme has been altered from this work by replacing thelearning system of the view layer with a variant of Fuzzy Simplified ART(F-SART; Baraldi and Alpaydin, 1998).

The View layer (280) clusters shroud-gated visual representations ofobject views according to their feature similarity. The Object layer(290) learns pose-invariant object representations by associatingdifferent view prototypes according to their temporal continuityprovided by the reset signal from the Where system. The Name layer (300)further groups different objects under the same user label if given froman optionally present Teacher (160). As an external teacher shapes theassociation between objects and their labels, this information is fedback from the Name layer to the Object and View layers to improve thespeed and accuracy of future object learning.

Disambiguation Map (310)

This section describes in detail the Disambiguation Map as introduced bySherbakov et al. (2013a, b). A single input view passed to the WhatSystem can activate multiple view, object, and name nodes. Although theoutput of each of these layers is sparse, the system output canoccasionally be unsure about object identity in the absence of anexternal teacher. This is called “object ambiguity”, as a single view ofan object can be associated with many objects.

To facilitate object disambiguation, OpenEye uses a novel, dynamicallyconstructed, disambiguation map that suggests potential saccade targetsto the Where Pathway that would maximally inform the What System as towhich of the potential object representations best matches the actualviewed object. This map compares those views within the resonant domainthat are activating disparate object categories, and activates thedisambiguation map in the areas where input and view prototypesdisagree.

The map is currently defined as the weighted average of the featuredifferences between the input {right arrow over (x)} and weighttemplates {right arrow over (W)}_(j) ^(xv), where each template is mixedonly if a view v_(j) in the resonant domain is coactive with itsassociated object category o_(j) ^(v). Specifically, the disambiguationmap {right arrow over (y)} is defined as

$\begin{matrix}{{\overset{\rightarrow}{\gamma} = \frac{\Sigma_{j}o_{j}^{v}v_{j}{{{\overset{\rightarrow}{W}}_{j}^{xv} - \overset{\rightarrow}{x}}}}{\Sigma_{j}o_{j}^{v}v_{j}}},} & (280.7)\end{matrix}$

where o_(j) ^(v) of is the feedback activity of the object categorylayer to a particular view category j, whose activities are given by

{right arrow over (o)} ^(v) =f _(∞)(W ^(vo) o ).  (280.7a)

Note that equation 280.7 could be modified to include multiple featuresby including a sum over features m.

Alternative OpenEye What System: stable sparse coding

FIG. 4 shows an alternative classifier architecture for the view layer280 and object layer 290 in FIG. 3. This alternative classifierarchitecture goes beyond the Fuzzy Simplified ART (f-sART; Baraldi andAlpaydin, 1998). FIG. 4 shows that this alternative classifier isorganized in several submodules that mimic some properties of therespective cortical and subcortical areas that they are named after(FIG. 4):

Thalamus layer (400)

Basal Forebrain (410)

Layer 4 (420)

Layer 2/3 (430).

As shown in FIG. 4, the thalamus layer 400 receives or obtains inputfrom a sensor—in this case, image data, such as still pictures or video,from the RGB sensor 100. The thalamus layer 400 pre-processes the sensordata and provides inputs to Layer 4 (420) and the Basal Forebrain (100).Layer 4 learns online to represent its input (400), whereas BasalForebrain (410) regulates learning in (420). Layer 2/3 (430) maps thelearned representation in Layer 4 (420) to discrete classes which can bedefined by a teacher (160)

One example of this system has been reduced to practice in a syntheticalphabet dataset comprised of an 28 by 28=784 input image, where 2N=1568is the total size of the input image in (100) after complement coding,R=400 is the number of coding cells (410), and M=26 is the size of layer2/3 (430).

Thalamus (400)

Unsupervised learning of a sparse encoder occurs in the synapses betweenthalamus (400) and Layer 4 (410). The sensor input, I, is processedthrough (400) as a shunting network, x, intended to represent theon-center off-surround processing in the thalamic relay cells. Theinputs (100) and Thalamus (400) are represented here as two-dimensionalfields with indices i and j, which run from 1 to n (28). The thalamus(400) includes two channels, light and dark, that receive the positiveinput and its complement, respectively. The activities are denoted x^(L)for the light channel and x^(D) for the dark channel:

$\begin{matrix}{{\frac{{dx}_{ij}^{L}}{dt} = {\left\lbrack {{{- \alpha_{x}}x_{ij}^{L}} + {\Sigma_{kl}{g_{i,j,k,l}\left( {B - x_{ij}^{L}} \right)}\left( I_{kl} \right)} - {{Dx}_{ij}^{L}\Sigma_{kl}{h_{i,j,k,l}\left( x_{kl}^{L} \right)}} + {{E\left( {C - x_{ij}^{L}} \right)}\Sigma \; w_{ijr}Y_{r}}} \right\rbrack \text{/}\tau_{x}}},\mspace{76mu} {and}} & (1) \\{{\frac{{dx}_{ij}^{D}}{dt} = {\left\lbrack {{{- \alpha_{x}}x_{ij}^{D}} + {\Sigma_{kl}{g_{i,j,k,l}\left( {B - x_{ij}^{D}} \right)}\left( {1 - I_{kl}} \right)} - {{Dx}_{ij}^{D}\Sigma_{kl}{h_{i,j,k,l}\left( x_{kl}^{D} \right)}} + {{E\left( {C - x_{ij}^{D}} \right)}\Sigma \; w_{ijr}Y_{r}}} \right\rbrack \text{/}\tau_{x}}},} & (2)\end{matrix}$

where α_(x)=0.4, B=1,C=−0.1,D=0.05, E=0.15, τ_(x)=6 and g is a Gaussianfunction of the difference between i and k centered around 0 (mean) witha width of 0.4 (the standard deviation), h is a likewise function of awidth 2.8. The parameter C corresponds to a hyperpolarization value.

Basal Forebrain (410)

The basal forebrain includes a single cell (b, a) which responds tofluctuations in the activities of thalamic cells (400) from bothchannels:

$\begin{matrix}{{\frac{db}{dt} = {\left\lbrack {{- b} + {G\mspace{14mu} \Sigma_{ij}\frac{x_{ij}}{2N}}} \right\rbrack \text{/}\tau_{b}}},{and}} & (3) \\{{a = {{\epsilon_{a} + {l\left( {{G\mspace{14mu} \Sigma_{ij}\frac{x_{ij}}{2N}} - b} \right)} + {H\mspace{14mu} \Sigma_{ij}{l\left( {{- \frac{x_{ij}}{N}} - V} \right)}}}}},} & (4)\end{matrix}$

where G=2, V=0.00001, H=1.8, τ_(b)=1/0.13, ∈_(a)=0.009, and a representsthe cholinergic signal. The function l is a threshold linear functionthat nullifies all negative values:

$\begin{matrix}{{l(u)} = \left\{ {\begin{matrix}{u,{u > 0}} \\{0,{u \leq 0}}\end{matrix}.} \right.} & (5)\end{matrix}$

This cholinergic signal drives a slower signal that biologically mightcorrespond to muscarinic receptor activation:

$\begin{matrix}{{\frac{d\overset{\_}{a}}{dt} = \frac{{- \overset{\_}{a}} + {Ja}}{\tau_{\overset{\_}{a}}}},{and}} & (6) \\{{\hat{a} = {{l\left( \overset{\_}{a} \right)} + \epsilon_{\overset{\_}{a}}}},} & (7)\end{matrix}$

where J=0.9, τ _(α) =20, and ∈ _(α) =0.001.

Layer 4 (420)

The activities at cortical coding cells in layer 4 (y) fluctuate basedon their thalamic input (x), lateral inhibition and modulation (420):

$\begin{matrix}{{\frac{y_{r}}{dt} = {\left\lbrack {{f_{y}\left( {{- t_{r}} + {\Sigma_{ij}w_{ijr}{x_{ij}}} + {\Sigma_{s}q_{sr}y_{s}}} \right)} - {y_{r}\left( {{C_{y}a} + a_{y}} \right)}} \right\rbrack \text{/}\tau_{y}}},} & (8)\end{matrix}$

where C_(y)=3.1, α_(y)=0.001, and τ_(y)=2.5. The signal function is alogistic function:

$\begin{matrix}{{{f_{y}(u)} = \frac{1}{1 + e^{- {\beta_{y}{(u)}}}}},} & (9)\end{matrix}$

where β_(y)=30. Layer 4 cortical activity must exceed a threshold beforeit affects learning or it is transmitted back to thalamus:

$\begin{matrix}{Y_{r} = \left\{ {\begin{matrix}{1,{y_{r} \geq 0.5}} \\{0,{y_{r} < 0.5}}\end{matrix}.} \right.} & (10)\end{matrix}$

Excitability homeostasis occurs at Layer 4 (y, Y) neurons by adaptivethresholds (t):

$\begin{matrix}{\frac{{dt}_{j}}{dt} = \frac{\left( {Y_{r} - \rho} \right)\left( {{C_{t}a} + \theta_{T}} \right)}{\tau_{t}}} & (11)\end{matrix}$

where C_(t)=0.6, θ_(T)=0.3, τ_(t)=1/0.06, and ρ is the sparsenessparameter, set to 0.05. The values for t_(j) are set to random valuesuniformly chosen from the range 0.5 to 2.3. The learning from Thalamicneurons (x, 400) to Layer 4 neurons (y, Y, 420) is governed by:

$\begin{matrix}{{\frac{w_{ijr}}{dt} = {{\lambda \left( {\left( {{x_{ij}} - w_{ijr}} \right)Y_{r}} \right)}\text{/}\tau_{w}}},} & (12)\end{matrix}$

where τ_(w)=384. The lateral inhibition (q) at Layer 4 (y, Y) isadaptive (an “anti-Hebbian” law):

$\begin{matrix}{{\frac{q_{rs}}{dt} = {{- {\lambda \left( {{Y_{r}Y_{s}} - \rho^{2}} \right)}}\text{/}\tau_{q}}},.} & (13)\end{matrix}$

where τ_(q)=167. The learning rate λ is a function of the strength offeedback and the ratio of cholinergic modulation:

$\begin{matrix}{{\lambda = {{L{{\left( {\Sigma_{ijr}\frac{4E\mspace{14mu} w_{ijr}Y_{r}}{2N}} \right) - 1}}} + {C_{\lambda}\frac{\hat{a}}{a}} + \epsilon_{\lambda}}},} & (14)\end{matrix}$

where L=0.12, C_(λ)=0.04, and ∈_(A)=0.005.

Layer 2/3 (430)

In this system, supervised learning of visual categories occurs in thesynapses between later 4 (420) and layer 2/3 (430). Specifically, layer2/3 neurons (z, 430) use a variant of the recurrent shunting on-centeroff-surround network:

$\begin{matrix}{{\tau_{z}\frac{{dz}_{m}}{dt}} = {{{- \alpha_{z}}z_{m}} + {\left( {B - z_{m}} \right)\left\lbrack {T_{m} + {f_{z}\left( z_{m} \right)} + {\frac{D_{z}}{R}{\sum\limits_{r}{\omega_{rm}Y_{r}}}}} \right\rbrack} - {{z_{m}\left\lbrack {{C_{z}\left( {1 - T_{m}} \right)} + {\sum\limits_{k}{f_{z}\left( z_{k} \right)}}} \right\rbrack}.}}} & (15)\end{matrix}$

where α_(y)=0.2, B=1, τ_(z)=1/0.75, D_(z)=120, C_(z)=0.1 and T is theaforementioned supervision, or Teacher (160), vector (i.e. the truth ortarget output). The supervision, therefore, acts as an input to theLayer 2/3 cells. It does not directly appear in the synaptic learninglaw. In this way, the full circuit is stackable in a hierarchical way.Higher cortical areas can simply send feedback that acts as thissupervision signal. The signal function is a logistic function:

$\begin{matrix}{{{f_{z}(u)} = \frac{1}{1 + e^{- {\beta_{z}{({u - \sigma_{z}})}}}}},} & (16)\end{matrix}$

where σ_(z)=0.65 and β_(y)=15, and corresponds to slope and threshold,respectively. The learning from Layer 4 neurons (y, Y) to Layer 2/3neurons (z) is governed by:

$\begin{matrix}{{\tau_{\omega}\frac{d\; \omega_{rm}}{dt}} = {{{\lambda_{\omega}\left( {\frac{Y_{r}}{R} - \omega_{rm}} \right)}{{f\left( z_{m} \right)}\left\lbrack {ɛ_{\omega} + {\sum\limits_{k}{f_{z}\left( z_{k} \right)}}} \right\rbrack}^{2}} + {\omega_{rm}\left( {\beta_{\omega} - {\sum\limits_{r}{\omega_{rm}z_{m}}}} \right)}}} & (17)\end{matrix}$

where λ_(ω)=0.6, β_(ω)=0.04 (400*0.0001), τ_(ω)=1/0.002, andε_(ω)=0.001.

CONCLUSION

While various inventive embodiments have been described and illustratedherein, those of ordinary skill in the art will readily envision avariety of other means and/or structures for performing the functionand/or obtaining the results and/or one or more of the advantagesdescribed herein, and each of such variations and/or modifications isdeemed to be within the scope of the inventive embodiments describedherein. More generally, those skilled in the art will readily appreciatethat all parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the inventive teachingsis/are used. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, many equivalentsto the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presentedby way of example only and that, within the scope of the appended claimsand equivalents thereto, inventive embodiments may be practicedotherwise than as specifically described and claimed. Inventiveembodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure.

The above-described embodiments can be implemented in any of numerousways. For example, embodiments of designing and making the technologydisclosed herein may be implemented using hardware, software or acombination thereof. When implemented in software, the software code canbe executed on any suitable processor or collection of processors,whether provided in a single computer or distributed among multiplecomputers.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in anysuitable form, including a local area network or a wide area network,such as an enterprise network, and intelligent network (IN) or theInternet. Such networks may be based on any suitable technology and mayoperate according to any suitable protocol and may include wirelessnetworks, wired networks or fiber optic networks.

The various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, various inventive concepts may be embodied as acomputer readable storage medium (or multiple computer readable storagemedia) (e.g., a computer memory, one or more floppy discs, compactdiscs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other non-transitory medium or tangible computer storagemedium) encoded with one or more programs that, when executed on one ormore computers or other processors, perform methods that implement thevarious embodiments of the invention discussed above. The computerreadable medium or media can be transportable, such that the program orprograms stored thereon can be loaded onto one or more differentcomputers or other processors to implement various aspects of thepresent invention as discussed above.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of embodiments as discussedabove. Additionally, it should be appreciated that according to oneaspect, one or more computer programs that when executed perform methodsof the present invention need not reside on a single computer orprocessor, but may be distributed in a modular fashion amongst a numberof different computers or processors to implement various aspects of thepresent invention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconvey relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Also, various inventive concepts may be embodied as one or more methods,of which an example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of.” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03.

REFERENCES

The following references are incorporated herein by reference in theirrespective entireties:

-   Adelson, E. H., Anderson, C. H., Bergen, J. R., Burt, P. J., &    Ogden, J. M. (1984). Pyramid methods in image processing. RCA    engineer, 29(6), 33-41.-   Baraldi, A. and Alpaydin, E. (1998). Simplified ART: A new class of    ART algorithms. International Computer Science Institute, Berkeley,    Calif., TR-98-004, 1998.-   Baraldi, A. and Alpaydin, E. (2002). Constructive feedforward ART    clustering networks—Part I. IEEE Transactions on Neural Networks    13(3), 645-661.-   Baraldi, A. and Parmiggiani, F. (1997). Fuzzy combination of    Kohonen's and ART neural network models to detect statistical    regularities in a random sequence of multi-valued input patterns. In    International Conference on Neural Networks, IEEE.-   Bengio, Y., Courville, A., & Vincent, P. (2013). Representation    learning: A review and new perspectives.-   Besl, P. J., & Jain, R. C. (1985). Three-dimensional object    recognition. ACM Computing Surveys (CSUR), 17(1), 75-145.-   Bradski, G., & Grossberg, S. (1995). Fast-learning VIEWNET    architectures for recognizing three-dimensional objects from    multiple two-dimensional views. Neural Networks, 8 (7-8), 1053-1080.-   Canny, J., A (1986) Computational Approach To Edge Detection, IEEE    Trans. Pattern Analysis and Machine Intelligence,    8(6):679-698.Carpenter, G. A. and Grossberg, S. (1987). A massively    parallel architecture for a self-organizing neural pattern    recognition machine. Computer Vision, Graphics, and Image Processing    37, 54-115.-   Carpenter, G. A., & Grossberg, S. (1987). A massively parallel    architecture for a self-organizing-   Carpenter, G. A., and Grossberg, S. (1995). Adaptive resonance    theory (ART). In M. Arbib (Ed.),-   The handbook of brain theory and neural networks. (pp. 79-82).    Cambridge, M.A.: MIT press.-   Carpenter, G. A., Grossberg, S. and Rosen, D. B. (1991). Fuzzy ART:    Fast stable learning and categorization of analog patterns by an    adaptive resonance system. Neural Networks 4, 759-771.-   Dosher, B. A., and Lu, Z. L. (2010). Mechanisms of perceptual    attention in precuing of location. Vision Res., 40(10-12).    1269-1292.-   Fazl, A., Grossberg, S., and Mingolla, E. (2009). View-invariant    object category learning, recognition, and search: How spatial and    object attention are coordinated using surface-based attentional    shrouds. Cognitive Psychology 58, 1-48.-   Földiák, P. (1990). Forming sparse representations by local    anti-Hebbian learning, Biological Cybernetics, vol. 64, pp. 165-170.-   Friston K., Adams R., Perrinet L., & Breakspear M. (2012).    Perceptions as hypotheses: saccades as experiments. Frontiers in    Psychology, 3 (151), 1-20.-   Geiger A, Lenz P, and Urtasun R (2012). Are we ready for autonomous    driving? The KITTI vision benchmark suite. In Computer Vision and    Pattern Recognition (CVPR), Providence, USA.-   George, D. and Hawkins, J. (2009). Towards a mathematical theory of    cortical micro-circuits. PLoS Computational Biology 5(10), 1-26.-   Grossberg, S. (1973). Contour enhancement, short-term memory, and    constancies in reverberating neural networks. Studies in Applied    Mathematics 52, 213-257.-   Grossberg, S., and Huang, T. R. (2009). ARTSCENE: A neural system    for natural scene classification. Journal of Vision, 9 (4), 6.1-19.    doi:10.1167/9.4.6-   Grossberg, S., and Versace, M. (2008) Spikes, synchrony, and    attentive learning by laminar thalamocortical circuits. Brain    Research, 1218C, 278-312 [Authors listed alphabetically].-   Hinton, G. E., Osindero, S., and Teh, Y. (2006). A fast learning    algorithm for deep belief nets. Neural Computation, 18, 1527-1554.-   Itti, L., and Koch, C. (2001). Computational modelling of visual    attention. Nature Reviews Neuroscience, 2 (3), 194-203.-   Itti, L., Koch, C., and Niebur, E. (1998). A Model of Saliency-Based    Visual Attention for Rapid Scene Analysis, 1-6.-   Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009,    September). What is the best multi-stage architecture for object    recognition?. In Computer Vision, 2009 IEEE 12th International    Conference on (pp. 2146-2153). IEEE.-   Kowler, E. (2011). Eye movements: The past 25 years. Vision    Research, 51(13), 1457-1483. doi:10.1016/j.visres.2010.12.014-   Larochelle H., & Hinton G. (2012). Learning to combine foveal    glimpses with a third-order Boltzmann machine. NIPS 2010, 1243-1251.-   LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010, May). Convolutional    networks and applications in vision. In Circuits and Systems    (ISCAS), Proceedings of 2010 IEEE International Symposium on (pp.    253-256). IEEE.-   Lee, D. D. and Seung, H. S. (1999). Learning the parts of objects by    non-negative matrix factorization. Nature, 401(6755):788-791.-   Lee, D. D., and Seung, H. S. (1997). “Unsupervised learning by    convex and conic coding.” Advances in Neural Information Processing    Systems, 9.-   Lowe, D. G. (2004). Distinctive Image Features from Scale-Invariant    Keypoints. Journal International Journal of Computer Vision archive    Volume 60, 2, 91-110.-   Lu, Z. L., Liu, J., and Dosher, B. A. (2010) Modeling mechanisms of    perceptual learning with augmented Hebbian re-weighting. Vision    Research, 50(4). 375-390.-   Mishkin M, Ungerleider L G. (1982). “Contribution of striate inputs    to the visuospatial functions of parieto-preoccipital cortex in    monkeys,” Behav Brain Res, 6 (1): 57-77.-   Najemnik, J., and Geisler, W. (2009). Simple summation rule for    optimal fixation selection in visual search. Vision Research. 49,    1286-1294. neural pattern-recognition machine. Computer Vision    Graphics and Image Processing, 37 (1), 54-115.-   Oja, E. (1982). Simplified neuron model as a principal component    analyzer. Journal of Mathematical Biology 15(3), 267-273.-   Pessoa L, Thompson E, and Noe A (1998). Finding out about    filling-in: A guide to perceptual completion for visual science and    the philosophy of perception. Behavioral and Brain Sciences 21(6),    723-748.-   Raijmakers, M. E. J., and Molenaar, P. (1997). Exact ART: A complete    implementation of an ART network Neural networks 10 (4), 649-669.-   Ranzato, M. A., Huang, F. J., Boureau, Y. L., & Lecun, Y. (2007,    June). Unsupervised learning of invariant feature hierarchies with    applications to object recognition. In Computer Vision and Pattern    Recognition, 2007. CVPR′07. IEEE Conference on (pp. 1-8). IEEE.-   Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object    recognition in cortex. Nature Neuroscience, 2 (11), 1019-1025.-   Riesenhuber, M., & Poggio, T. (2000). Models of object recognition.    Nature neuroscience, 3, 1199-1204.-   Rublee E, Rabaud V, Konolige K, and Bradski G (2011). ORB: An    efficient alternative to SIFT or SURF. In IEEE International    Conference on Computer Vision (ICCV), 2564-2571.-   Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011). ORB: An    efficient alternative to SIFT or SURF. In IEEE International    Conference on Computer Vision (ICCV) 2011, 2564-2571.-   Rumelhart D., Hinton G., and Williams, R. (1986). Learning internal    representations by error propagation. In Parallel distributed    processing: explorations in the microstructure of cognition, vol. 1,    MIT Press.-   Russell B, Torralba A, Murphy K, and Freeman W T (2008). LabelMe: A    database and web-based tool for image annotation. International    Journal of Computer Vision 77(1-3), 157-173.-   Salakhutdinov, R., & Hinton, G. E. (2009). Deep boltzmann machines.    In International Conference on Artificial Intelligence and    Statistics (pp. 448-455).-   Seibert, M., & Waxman, A. M. (1992). Adaptive 3-D Object Recognition    from Multiple Views. IEEE Transactions on Pattern Analysis and    Machine Intelligence, 14 (2), 107-124.-   Sherbakov, L., Livitz, G., Sohail, A., Gorchetchnikov, A., Mingolla,    E., Ames, H., and Versace, M. (2013a) CogEye: An online active    vision system that disambiguates and recognizes objects. NeuComp    2013.-   Sherbakov, L., Livitz, G., Sohail, A., Gorchetchnikov, A., Mingolla,    E., Ames, H., and Versace, M (2013b) A computational model of the    role of eye-movements in object disambiguation. Cosyne, Feb. 28-Mar.    3, 2013. Salt Lake City, Utah, USA.-   Smolensky, P. (1986). Information processing in dynamical systems:    Foundations of harmony theory. In D. E. Rumelhartand J. L.    McClelland, editors, Parallel Distributed Processing, volume 1,    chapter 6, pages 194-281. MIT Press, Cambridge.-   Spratling, M. W. (2008). Predictive coding as a model of biased    competition in visual attention. Vision Research, 48(12):1391-1408.-   Spratling, M. W. (2012). Unsupervised learning of generative and    discriminative weights encoding elementary image components in a    predictive coding model of cortical function. Neural Computation,    24(1):60-103.-   Spratling, M. W., De Meyer, K., and Kompass, R. (2009). Unsupervised    learning of overlapping image components using divisive input    modulation. Computational intelligence and neuroscience.-   Tong, F., Ze-Nian Li, (1995). Reciprocal-wedge transform for    space-variant sensing,”Pattern Analysis and Machine Intelligence,    IEEE Transactions on, vol. 17, no. 5, pp. 500-51. doi:    10.1109/34.391393-   Torralba, A., Oliva, A., Castelhano, M. S., Henderson, J. M. (2006).    Contextual guidance of eye movements and attention in real-world    scenes: the role of global features in object search. Psychological    Review, 113(4).766-786.-   Versace, M. (2006) From spikes to interareal synchrony: how    attentive matching and resonance control learning and information    processing by laminar thalamocortical circuits. NSF Science of    Learning Centers PI Meeting, Washington, D.C., USA.-   Webster, Bachevalier, Ungerleider (1994). Connections of IT areas    TEO and TE with parietal and frontal cortex in macaque monkeys.    Cerebal Cortex, 4(5), 470-483.

1. A method comprising: acquiring a plurality of images of at least a portion of an environment with an image sensor; constructing a spatial shroud fitting a form of an object in a first image of the plurality of images from the first image; identifying the object in the first image based at least in part on the spatial shroud; translating and/or transforming the spatial shroud based at least in part on a change in a position and/or an orientation of the image sensor; determining if the spatial shroud fits the form of the object in a second image in the plurality of images; and in response to determining that the spatial shroud fits the form of the object, identifying a position of the object in the environment based at least in part on the spatial shroud. 